Touch event model programming interface

ABSTRACT

A method includes receiving a document from a server, where the document includes an embedded script. The method also includes rendering and displaying the document at an electronic device and executing the embedded script. The method furthermore includes establishing a touchevent interface object that includes a plurality of touchlists, and upon detecting one or more touches on the touch-sensitive surface, updating the touchevent interface object with touch data, including values in two or more of the touchlists. The method includes further executing the embedded script in accordance with the values in at least one of the two or more touchlists.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/042,299, filed Mar. 4, 2008, entitled “Touch Event Model ProgrammingInterface” which is herein incorporated by reference in its entirety.

The subject matter of this application is related to application Ser.No. 12/042,237, for “Touch Event Processing for Web Pages,” filed Mar.4, 2008, and application Ser. No. 12/042,067, for “Touch Event Model forWeb Pages,” filed Mar. 4, 2008, the contents of which are hereinincorporated by reference.

TECHNICAL FIELD

This subject matter is generally related to web browsing services.

BACKGROUND

Web pages are created using a markup language which provides a means todescribe a structure of text-based information in a document and tosupplement that text with interactive forms, embedded images, and otherobjects. One popular markup language is HyperText Markup Language (HTML)which is written in the form of tags surrounded by angle brackets. HTMLcan describe the appearance and semantics of a web page, and can includeembedded scripting language code (e.g., JavaScript®) which can affectthe behavior of web browsers and other HTML processors. JavaScript®provides developers with the ability to add mouse event handlers orevent listeners in a web page. These mouse event handlers can beassigned to specific regions of the web page and configured to receivemouse events in those regions, such as mouse up or mouse down events.

By contrast, web pages that are navigated with a touch sensitive deviceoften need to respond to touch events generated by a user touching a webpage with one or more fingers and making gestures. Conventional mouseevent handlers cannot correctly interpret these touch events. Thus touchevents require a different touch event model to correctly interprettouch events and to allow developers to fully utilize the capabilitiesof a touch sensitive display or device.

SUMMARY

One or more touch input signals can be obtained from a touch sensitivedevice. A touch event model can be used to determine touch and/orgesture events based on the touch input signals. The touch and gestureevents can be associated with touch input signals generated fromdifferent regions of a web page displayed on the touch sensitive device.Access can be provided to at least one touch or gesture event through aprogramming interface.

In some implementations, in a web browser, a method comprising:receiving a rotation value associated with a gesture event; anddynamically rotating an element of a web page associated with thegesture event in the web browser based on the rotation value, whereinthe rotation value is a relative delta in degrees. The gesture event canincludes two or more touch events.

In some implementations, in a web browser a method includes: receiving ascaling value associated with a gesture event; dynamically resizing anelement of a web page associated with the gesture event in the webbrowser based on the scaling value, wherein the scaling value is arelative delta in document pixels. The gesture event can be associatedwith two or more touch events.

In some implementations, in a web browser, a method includes: receivinga touch list, the touch list including touch data to identify one ormore touches on a web page, wherein the touch data includes a touchidentifier and at least one set of touch location coordinates, whereinthe touch list further includes data to a touch event target associatedwith each touch, wherein the at least one set of touch locationcoordinates includes a set of client coordinates, a set of pagecoordinates, and a set of screen coordinates. The touch data canidentify one or more changed touches.

Other implementations are disclosed which are directed to systems,methods and computer-readable mediums.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example web page document.

FIG. 2 illustrates a processing stack of an example multi-touch capabledevice.

FIG. 3 is a flow diagram of an example process for processing touchevents.

FIG. 4 illustrates an example multi-touch capable device.

FIG. 5 is a block diagram of an example network operating environmentfor the multi-touch capable device of FIG. 4.

FIG. 6 is a block diagram of an example implementation of themulti-touch capable device of FIG. 4.

DETAILED DESCRIPTION Example Web Page Structure and DOM

FIG. 1A shows an example web page 100 that can be displayed on abrowser. The browser may be hosted on a portable device, such as themulti-touch capable device 400 of FIG. 4. One or more elements, element102 (“element 1”), element 104 (“element 2”), and element 106 (“element3”) can be displayed on the web page 100. The elements 102, 104, 106 cancorrespond to regions of the web page 100 that a user can select andadditional functionality can be provided as a result of the selection.The elements can, for example, correspond to buttons on the web page100. The elements can also be nested so that one element containsanother element. For example, element 104 contains element 108. In theexample shown, the element 108 is a scrubber control nested withinelement 104 which can be a media player user interface, for example.

In some implementations, a user may perform various functions with theelements on the web page 100 using a finger instead of a mouse. The usercan, for example, touch the elements of the web page 100 using the touchsensitive display 402 shown in FIG. 4. In one example, the user canselect an element by touching the element with one or more fingersand/or making a gesture, such as a swiping, pinching or rotating motion.To recognize touch input signals, certain areas of the web page 100 maybe associated with touch event handlers. This can be accomplished with aDOM and embedded scripting language, as will be described in referenceto FIG. 1B.

FIG. 1B is an exemplary DOM 150 associated with the web page 100. TheDOM 150 provides a structural representation of the web page 100 anddescribes the web page content as a set of objects that a scriptinglanguage (e.g., JavaScript®) can interpret. In some implementations, theDOM 150 provides access to the structure of a web page by mapping theelements 102, 104, 106, 108 in the web page 100 to individual nodes of atree. For example, element 102 corresponds to node 154. Element 104corresponds to node 156. Element 106 corresponds to node 158. Element108 corresponds to node 160. The root node 152 corresponds to theoverall web page 100.

In some implementations, one or more elements 102, 104, 106, 108 in webpage 100 can be associated with one or more corresponding touch eventhandler by associating corresponding nodes in the DOM 150 with the touchevent handlers. Touch event handlers can be inserted into HTML tags ofthe web page 100 and can execute a scripting language to perform anaction when, for example, a user touches or gestures within one of theelements on the web page 100. For example, JavaScript® can work with theDOM 150 to attach actions to different touch events.

In some implementations, one or more elements 102, 104, 106, 108 canreceive touch input which is detected by an event handler or listener.The touch input can be detected and processed into touch events by atouch event model which can be implemented in one or more layers of asoftware stack, as described in reference to FIG. 2. The touch eventscan be further processed by the web page 100. The touch events can be ina format (e.g., attributes) that are easier to use in an applicationthan raw touch input signals generated by the touch sensitive device.For example, each touch event can include a set of coordinates at whicha touch is currently occurring.

Each element in the web page 100, and its associated event handler, canreceive, process and handle touch events. For example, if driver 202(FIG. 2) senses a touch point 110 associated with element 102, or atouch point 112 associated with element 104, the event handlersassociated with elements 102 or 104 can each receive a separate touchevent indicating that the element has been touched and can optionallysend the touch event to the web page 100 for further processing. In someimplementations, if a region of the web page 100 is touched that doesnot correspond to an event handler, a browser in the applications layer214 can process the input, instead of the web page 100.

In some implementations, a touch event can be detected per finger pernode in the DOM 150. For example, a user can touch the touch sensitivedisplay 402 at touch point 110 and touch point 112 at substantially thesame time, and two separate touch events can be detected by the touchevent model. A separate touch event can be detected for touch point 110and touch point 112 because each node 102 and node 104 in the DOM 150are associated with a separate touch event handler.

In some implementations, touch events can be delivered to the web page100 as EventTargets. Some examples of touch events can includetouchstart, touchmove, touchend, and touchcancel. Other touch events arepossible. Touchstart is a touch event that is detected when a user firstplaces a finger on the touch sensitive display 402 in a region on theweb page 100 that is associated with an event handler. When the usermoves his finger around on the web page 100, one or more touchmoveevents can be detected. When the user raises his finger off of the webpage 100, a touchend event is detected. The touchcancel can be detectedwhen the system interrupts regular event handling. For example, atouchcancel event can occur when the touch sensitive display 402 islocked out to prevent inadvertent touches.

In some implementations, gesture events can also be detected bycombining two or more touch events. Like touch events, the gestureevents (GestureEvents) can also be delivered to the web page 100 asEventTargets. Some examples of gesture events can be gesturestart,gesturechange, and gestureend. The gesture events can contain scaleand/or rotation information. The rotation information can include arotation value that is a relative delta in degrees. An element on thewebpage 100 can be dynamically rotated base on the rotation value. Thescale information can include a scaling value that is a relative deltain document pixels. An element on the webpage 100 associated with thegesture event can be dynamically resized based on the scaling value.Other gesture events are possible.

In some implementations, a touch list can be received that includestouch event data to identify one or more touches on the web page 100.The touch event data can include a touch identifier and at least one setof touch location coordinates. The touch list can also include touchevent data to a touch event target associated with each touch. In someimplementations, the one set of touch location coordinates can includeclient coordinates, page coordinates, and screen coordinates. In someimplementations, the touch event data can identify one or more changedtouches.

In some implementations, GestureEvents can be sent to the web page 100before TouchEvents. For example, if a user places fingers on touch point110 and touch point 112, then makes a rotating gesture with thosefingers clockwise or counterclockwise on the touch sensitive display,the touch event model detects these multiple touch events and combinesthe touch events into a gesture event. The gesture event can then besent to the web page 100, followed by the touch events that werecombined to form the gesture event. This way, a developer has access toa gesture event and the individual touch events of the gesture eventwhich provides the developer with more flexibility when developing a webapplication.

In some implementations, the touch events are received in the followingorder: a touchstart event, one or more touchmove events and a touchendor touchcancel event. Using the example of FIG. 1A, when the usertouches touch point 110, a first touchstart event is detected by a firsttouch event handler associated with element 102. When the user touchestouch point 112, a second touchstart event is detected by a second touchevent handler associated with element 104. As the user rotates herfingers without lifting her fingers, the first and second touch eventhandlers detect touchmove events which can be interpreted by the touchevent model as a rotation gesture event. When the user finishes therotation and lifts her fingers from the web page 100, the first andsecond touch event handlers detect touchend events. All or some of thesetouch events can be made available to developers through a touch eventApplication Programming Interface (API). The touch API can be madeavailable to developers as a Software Development Kit (SDK) or as partof an application (e.g., as part of a browser tool kit). The touch eventAPI can rely on other services, frameworks and an operating system toperform its various functions. These services, frameworks and operatingsystem can be part of a software or processing stack, as described inreference to FIG. 2, where the touch events are associated withattributes that can be inserted in documents to define event actions inan application.

Example IDL

An example touch event model will now be described in InterfaceDescription Language (IDL). The functionality and data structures of theIDL can be accessed through an API by a web designer or applicationdeveloper. Access to touch events and/or gesture events can beassociated with attributes that can be inserted in a markup languagedocument (e.g., HTML, XML) to define event actions in an application.For example, the attributes can be inserted in one or more HTML tags inan HTML document for generating the web page displayed on the touchsensitive display 402. The event actions can include running an embeddedscript (e.g., JavaScript®).

interface [  Conditional=TOUCH_EVENTS,  GenerateConstructor ] TouchEvent: UIEvent {  void initTouchEvent(in AtomicString type,    in booleancanBubble,    in boolean cancelable,    in DOMWindow view,    in longdetail,    in long screenX,    in long screenY,    in long clientX,   in long clientY,    in boolean ctrlKey,    in boolean altKey,    inboolean shiftKey,    in boolean metaKey,    in TouchList touches,    inTouchList targetTouches,    in TouchList changedTouches,    in longscale,    in long rotation);  readonly attribute TouchList touches; //all touches  readonly attribute TouchList targetTouches; // all touchesin this TouchEvent Target  readonly attribute TouchListchangedTouches; // all touches changed in the current event  readonlyattribute long scale;  readonly attribute long rotation;  readonlyattribute boolean ctrlKey;  readonly attribute boolean shiftKey; readonly attribute boolean altKey;  readonly attribute boolean metaKey;}; interface [  Conditional=TOUCH_EVENTS, ] Touch {  readonly attributeEventTarget target;  readonly attribute long identifier;  readonlyattribute long clientX;  readonly attribute long clientY;  readonlyattribute long pageX;  readonly attribute long pageY;  readonlyattribute long screenX;  readonly attribute long screenY; }; interface [ Conditional=TOUCH_EVENTS,  HasIndexGetter, ] TouchList {  readonlyattribute unsigned long length;  Touch  item(in unsigned long index); };interface [  Conditional=TOUCH_EVENTS,  GenerateConstructor ]GestureEvent : UIEvent {  void initGestureEvent( in AtomicString type,   in boolean canBubble,    in boolean cancelable,    in DOMWindow view,   in long detail,    in long screenX,    in long screenY,    in longclientX,    in long clientY,    in boolean ctrlKey,    in booleanaltKey,    in boolean shiftKey,    in boolean metaKey,    in EventTargettarget,    in long scale,    in long rotation);  readonly attributeEventTarget target;  readonly attribute long scale;  readonly attributelong rotation;  readonly attribute boolean ctrlKey;  readonly attributeboolean shiftKey;  readonly attribute boolean altKey;  readonlyattribute boolean metaKey; }; In Document.id1; Touch createTouch(inEventTarget target,    in long identifier,    in long clientY,    inlong clientX,    in long pageX,    in long pageY,    in long screenX,   in long screenY)  raises (DOMException); [Custom]TouchList createTouchList()  raises (DOMException);

The following is example of HTML code snippet for processing touchevents using the example IDL above. The HTML below shows, for example,that the touch event listener TouchStart and GestureStart were added toan element with the HTML code:

this.element.addEventListener(‘touchstart’, function(e) {return self.onTouchStart(e)}, false);

this.element.addEventListener(‘gesturestart’, function(e) {returnself.on GestureStart(e)}, false);

The HTML code corresponding to the IDL above may be as follows:

<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.01 Transitional//EN”“http://www.w3.org/TR/html4/loose.dtd”> <html lang=“en”>  <head>   <metahttp-equiv=“Content-Type” content=“text/html; charset=utf-8”>    <metaname=“viewpoint” content=“initial-scale=1.0” />    <title>TransformGestures</title>    <style type=“text/css” media=“screen”>     .box {     position: absolute;      height: 150px;      width: 150px;     background-color: blue;     }     .box:active {     background-color: red;     }     body {      margin: 0px;     }    #container {      position: absolute;      width: 100%;      height:100%;     }     #main-box2 {      top: 10px;      left: 155px;     background: red;      z-index: 1;     }    </ style>    <scripttype=“text/javascript” charset=“utf-8”>     var trackedObjectCount = 0;    function Box(inElement)     {      var self = this;     this.element = inElement;      this.scale = 1.0;      this.rotation= 0;      this.position = ‘0,0’;     this.element.addEventListener(‘touchstart’, function(e) { returnself.onTouchStart(e) }, false);     this.element.addEventListener(‘gesturestart’, function(e) { returnself.onGestureStart(e) }, false);     }     Box.prototype = {      //position strings are “x,y” with no units      get position()      {      return this._position;      },      set position(pos)      {      this._position = pos;       var components = pos.split(‘,’)      var x = components[0];       var y = components[1];       constkUseTransform = true;       if (kUseTransform) {       this.element.style.webkitTransform = ‘rotate(’ + this.rotation +‘deg) scale(’ + this.scale + ‘) translate(’ + x + ‘px,’ + y + ‘px)’;      }       else {        this.element.style.left = x + ‘px’;       this.element.style.top = y + ‘px’;       }      },      get x()     {       return parseInt(this._position.split(‘,’)[0]);      },     set x(inX)      {       var comps = this._position.split(‘,’);      comps[0] = inX;       this.position = comps.join(‘,’);      },     get y()      {       return parseInt(this._position.split(‘,’)[1]);     },      set y(inY)      {       var comps =this._position.split(‘,’);       comps[1] = inY;       this.position =comps.join(‘,’);      },      filterEvent: function(e)      {       //Prevent the browser from doing its default thing (scroll, zoom)      e.preventDefault();       // Event listeners are added at thedocument level, so we receive gesturechange events for other elements.      return (e.target == this.element);      },      onTouchStart:function(e)      {       if (!this.filterEvent(e))        return false;      // Start tracking when the first finger comes down in this element      if (e.targetTouches.length != 1)        return false;      this.startX = e.targetTouches[0].clientX;       this.startY =e.targetTouches[0].clientY;       var self = this;       if(!(“touchMoveHandler” in this)) {        this.touchMoveHandler =function(e) { return self.onTouchMove(e) }        this.touchEndHandler =function(e) { return self.onTouchEnd(e) }       }      document.addEventListener(‘touchmove’, this.touchMoveHandler,false);       document.addEventListener(‘touchend’,this.touchEndHandler, false);       trackedObjectCount++;       returnfalse;      },      onTouchMove: function(e)      {       if(!this.filterEvent(e))        return false;       // Don't track motionwhen multiple touches are down in this element (that's a gesture)      if (e.targetTouches.length != 1)        return false;       varleftDelta = e.targetTouches[0].clientX − this.startX;       var topDelta= e.targetTouches[0].clientY − this.startY;       var newLeft =(this.x) + leftDelta;       var newTop = (this.y) + topDelta;      this.position = newLeft + ‘,’ + newTop;       this.startX =e.targetTouches[0].clientX;       this.startY =e.targetTouches[0].clientY;       return false;      },      onTouchEnd:function(e)      {       if (!this.filterEvent(e))        return false;      // Stop tracking when the last finger is removed from this element      if (e.targetTouches.length > 0)        return false;      document.removeEventListener(‘touchmove’, this.touchMoveHandler,false);       document.removeEventListener(‘touchend’,this.touchEndHandler, false);       trackedObjectCount−−;       returnfalse;      },      onGestureStart: function(e)      {       if(!this.filterEvent(e))        return false;       var self = this;      if (!(“gestureChangeHandler” in this)) {       this.gestureChangeHandler = function(e) { returnself.onGestureChange(e) }        this.gestureEndHandler = function(e) {return self.onGestureEnd(e) }       }      document.addEventListener(‘gesturechange’,this.gestureChangeHandler, true);      document.addEventListener(‘gestureend’, this.gestureEndHandler,true);       return false;      }      onGestureChange: function(e)     {       if (!this.filterEvent(e))        return false;       //Only interpret gestures when tracking one object. Otherwise, interpretraw touch events       // to move the tracked objects.       if(trackedObjectCount == 1) {        this.scale += e.scaling * 0.01;       this.rotation += e.rotation / 2;        this.position =this.position;       }       return false;      },      onGestureEnd:function(e)      {       if (!this.filterEvent(e))        return false;      document.removeEventListener(‘gesturechange’,this.gestureChangeHandler, true);      document.removeEventListener(‘gestureend’, this.gestureEndHandler,true);       return false;      },     }     function loaded()     {     new Box(document.getElementById(‘main-box’));      newBox(document.getElementById(‘main-box2’));     }    window.addEventListener(‘load’, loaded, true);    </script>  </head>  <body>   <div id=“container”>    <div id=“main-box”class=“box”></div>    <div id=“main-box2” class=“box”></div>   </div> </body> </html>

Example Processing Stack for Multi-Touch Device

FIG. 2 is a diagram of a processing stack of an example multi-touchcapable device. The touch event model described above can be implementedin one or more regions of the processing stack and user variousresources in the stack. The hardware 200 layer can include varioushardware interface components, such as a touch sensitive or enableddevice or touch sensitive display. The touch sensitive device caninclude a display and a panel that senses multiple touchessimultaneously. The hardware layer 200 can also include an accelerometerfor detecting an orientation of the touch sensitive display or device(e.g., portrait, landscape). Thus signals indicative of orientation canbe used by the touch event model to scale web pages for optimum display.

One or more drivers in a driver layer 202 can communicate with thehardware 200. For example, the drivers can receive and process touchinput signals generated by the touch sensitive display or device in thehardware layer 200. A core Operating System (OS) 204 can communicatewith the driver(s). The core OS 204 can process raw input data receivedfrom the driver(s). In some embodiments, the drivers can be consideredto be a part of the core OS 204.

A set of OS application programming interfaces (APIs) 206 cancommunicate with the core OS 204. These APIs can be a set of APIs thatare usually included with operating systems (such as, for example, Linuxor UNIX APIs). A set of core foundation APIs 208 can utilize the OS APIs206, and a set of foundation APIs 210 can utilize the core foundationAPIs 208.

Web page software development kit (SDK) 210 can include a set of APIsdesigned for use by applications running on the device. The touch eventAPIs can, for example, be included in the Web page SDK 210. The APIs ofthe Web page SDK 210 can utilize the foundation APIs 208. The Web pageSDK 210 can, for example, include Web KIT provided by Apple Inc. The Webpage SDK 210 can be offered as an API or can be accessible through anapplication, for example, a browser such as SAFARI®, provided by AppleInc.

Applications 214 running on the device can utilize the APIs of the Webpage SDK 210 to create web pages. The APIs of the Web page SDK 210 can,in turn, communicate with lower level elements, ultimately communicatingwith the touch sensitive display or device and various other userinterface hardware. While each layer can utilize the layer underneathit, that is not always required. For example, in some embodiments,applications 214 can occasionally communicate with OS APIs 206.

Example Touch Event Process

FIG. 3 is a flow diagram of a process 300 for providing access to touchand/or gesture events though an API. The process 300 can begin byobtaining one or more touch input signals (302). The touch input signalscan be obtained from a touch sensitive display or device. Adetermination of touch events and/or gestures can be made based on thetouch input signals using a touch event model (304). The touch eventscan be associated with regions of a web page displayed on a touchsensitive display or device. For example, the touch sensitive displaycan be a display on a mobile phone, and a touch sensitive device can bea touch sensitive pad on a notebook computer. Access to touch eventsand/or gesture events can be provided through a programming interface(306). For example, with respect to the HTML snippet described abovewith reference to FIG. 2, the snippet can be inserted into an HTMLdocument by the web developer to provides the developer with access totouch and/or gesture events. The touch events and/or gesture events canbe further processed by code in the HTML document to initiate eventactions (306).

Mobile Device Overview

FIG. 4 is a block diagram of an example multi-touch capable device 400.In some implementations, the multi-touch capable device 400 includes atouch sensitive display 402. The touch sensitive display 402 canimplement liquid crystal display (LCD) technology, light emittingpolymer display (LPD) technology, or some other display technology. Thetouch sensitive display 402 can be sensitive to haptic and/or tactilecontact with a user.

In some implementations, the touch sensitive display 402 can comprise amulti-touch sensitive display 402. A touch sensitive display 402 can,for example, process multiple simultaneous touch points, includingprocessing data related to the pressure, degree and/or position of eachtouch point. Such processing facilitates gestures and interactions withmultiple fingers, chording, and other interactions. Other touchsensitive display technologies can also be used, e.g., a display inwhich contact is made using a stylus or other pointing device. Someexamples of multi-touch sensitive display technology are described inU.S. Pat. Nos. 6,323,846, 6,570,557, 6,677,932, and U.S. PatentPublication 2002/0015024A1, each of which is incorporated by referenceherein in its entirety. In some implementations, the multi-touch capabledevice 400 can display one or more graphical user interfaces on thetouch sensitive display 402 for providing the user access to varioussystem objects and for conveying information to the user.

Example Multi-Touch Capable Device Functionality

In some implementations, the multi-touch capable device 400 canimplement multiple device functionalities, such as a telephony device,an e-mail device, a network data communication device, a Wi-Fi basestation device, and a media processing device. In some implementations,the multi-touch capable device 400 can include a web browser 404 fordisplaying web pages (e.g., web page 100). The touch sensitive display402 can receive touch input signals made on the web page 100 and thetouch model described above can be used to determine touch and/orgesture events based on the touch input signals. In someimplementations, the multi-touch capable device 400 can implementnetwork distribution functionality. In some implementations, the touchsensitive display 402 can be locked down when the multi-touch capabledevice 400 is proximate to the user's ear. This lockdown would cause atouchcancel event as described in reference to FIG. 1B.

In some implementations, an accelerometer 472 can be utilized to detectmovement of the multi-touch capable device 400, as indicated by thedirectional arrow 474. Accordingly, display objects and/or media can bepresented according to a detected orientation, e.g., portrait orlandscape. In some implementations, the multi-touch capable device 400may include circuitry and sensors for supporting a location determiningcapability, such as that provided by the global positioning system (GPS)or other positioning systems (e.g., systems using Wi-Fi access points,television signals, cellular grids, Uniform Resource Locators (URLs)).In some implementations, a positioning system (e.g., a GPS receiver) canbe integrated into the multi-touch capable device 400 or provided as aseparate device that can be coupled to the multi-touch capable device400 through an interface to provide access to location-based services.The multi-touch capable device 400 can also include one or more wirelesscommunication subsystems.

In some implementations, a port device, e.g., a Universal Serial Bus(USB) port, or a docking port, or some other wired port connection, canbe included. The port device can, for example, be utilized to establisha wired connection to other computing devices, such as other multi-touchcapable devices 400, network access devices, a personal computer, aprinter, or other processing devices capable of receiving and/ortransmitting data. In some implementations, the port device allows themulti-touch capable device 400 to synchronize with a host device usingone or more protocols, such as, for example, the TCP/IP, HTTP, UDP andany other known protocol.

Network Operating Environment

FIG. 5 is a block diagram of an example network operating environment600 for the multi-touch capable device 400 of FIG. 4. The multi-touchcapable device 400 of FIG. 4 can, for example, communicate over one ormore wired and/or wireless networks 510 in data communication. Forexample, a wireless network 512, e.g., a cellular network, cancommunicate with a wide area network (WAN) 514, such as the Internet, byuse of a gateway 516. Likewise, an access point 518, such as an 802.11gwireless access point, can provide communication access to the wide areanetwork 514. In some implementations, both voice and data communicationscan be established over the wireless network 512 and the access point518. For example, the multi-touch capable device 400 a can place andreceive phone calls (e.g., using VoIP protocols), send and receivee-mail messages (e.g., using POP3 protocol), and retrieve electronicdocuments and/or streams, such as web pages, photographs, and videos,over the wireless network 512, gateway 516, and wide area network 514(e.g., using TCP/IP or UDP protocols). Likewise, the multi-touch capabledevice 400 b can place and receive phone calls, send and receive e-mailmessages, and retrieve electronic documents over the access point 518and the wide area network 514. In some implementations, the multi-touchcapable device 400 can be physically connected to the access point 518using one or more cables and the access point 518 can be a personalcomputer. In this configuration, the multi-touch capable device 400 canbe referred to as a “tethered” device.

The multi-touch capable devices 400 a and 400 b can also establishcommunications by other means. For example, the multi-touch capabledevice 400 a can communicate with other wireless devices, e.g., othermulti-touch capable devices 400, cell phones, etc., over the wirelessnetwork 512. Likewise, the multi-touch capable device 400 a and 400 bcan establish peer-to-peer communications 520, e.g., a personal areanetwork, by use of one or more communication subsystems, such as theBluetooth™ communication device 488 shown in FIG. 4. Other communicationprotocols and topologies can also be implemented.

The multi-touch capable device 400 can, for example, communicate with anetwork resource 530 over the one or more wired and/or wireless networks510. For example, the network resource can be a web server fordelivering web pages which can be touched via the touch model, asdescribed in reference to FIGS. 1-2.

Other services can also be provided, including a software update servicethat automatically determines whether software updates exist forsoftware on the multi-touch capable device 400, then downloads thesoftware updates to the multi-touch capable device 400 where it can bemanually or automatically unpacked and/or installed.

Example Mobile Device Architecture

FIG. 6 is a block diagram 600 of an example implementation of themulti-touch capable device 400 of FIG. 4. The multi-touch capable device400 can include a memory interface 602, one or more data processors,image processors and/or central processing units 604, and a peripheralsinterface 606. The memory interface 602, the one or more processors 604and/or the peripherals interface 606 can be separate components or canbe integrated in one or more integrated circuits. The various componentsin the multi-touch capable device 400 can be coupled by one or morecommunication buses or signal lines.

Sensors, devices and subsystems can be coupled to the peripheralsinterface 606 to facilitate multiple functionalities. For example, amotion sensor 610, a light sensor 612, and a proximity sensor 614 can becoupled to the peripherals interface 606 to facilitate the orientation,lighting and proximity functions described with respect to FIG. 4. Othersensors 616 can also be connected to the peripherals interface 606, suchas a positioning system (e.g., GPS receiver), a temperature sensor, abiometric sensor, or other sensing device, to facilitate relatedfunctionalities.

A camera subsystem 620 and an optical sensor 622, e.g., a chargedcoupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)optical sensor, can be utilized to facilitate camera functions, such asrecording photographs and video clips.

Communication functions can be facilitated through one or more wirelesscommunication subsystems 624, which can include radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of thecommunication subsystem 624 can depend on the communication network(s)over which the multi-touch capable device 400 is intended to operate.For example, a multi-touch capable device 400 may include communicationsubsystems 624 designed to operate over a GSM network, a GPRS network,an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth™ network. Inparticular, the wireless communication subsystems 624 may includehosting protocols such that the device 500 may be configured as a basestation for other wireless devices.

An audio subsystem 626 can be coupled to a speaker 628 and a microphone630 to facilitate voice-enabled functions, such as voice recognition,voice replication, digital recording, and telephony functions.

The I/O subsystem 640 can include a touch screen controller 642 and/orother input controller(s) 644. The touch-screen controller 642 can becoupled to a touch screen 646. The touch screen 646 and touch screencontroller 642 can, for example, detect contact and movement or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies, as well as other proximity sensorarrays or other elements for determining one or more points of contactwith the touch screen 646.

The other input controller(s) 644 can be coupled to other input/controldevices 648, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port, and/or a pointer device such as a stylus. Theone or more buttons (not shown) can include an up/down button for volumecontrol of the speaker 628 and/or the microphone 630.

In one implementation, a pressing of the button for a first duration maydisengage a lock of the touch screen 646; and a pressing of the buttonfor a second duration that is longer than the first duration may turnpower to the multi-touch capable device 400 on or off. The user may beable to customize a functionality of one or more of the buttons. Thetouch screen 646 can, for example, also be used to implement virtual orsoft buttons and/or a keypad or keyboard.

In some implementations, the multi-touch capable device 400 can presentrecorded audio and/or video files, such as MP3, AAC, and MPEG files. Insome implementations, the multi-touch capable device 400 can include thefunctionality of an MP3 player, such as an iPod™. The multi-touchcapable device 400 may, therefore, include a 32-pin connector that iscompatible with the iPod. Other input/output and control devices canalso be used.

The memory interface 602 can be coupled to memory 650. The memory 650can include high-speed random access memory and/or non-volatile memory,such as one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR). The memory 650can store an operating system 652, such as Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or an embedded operating system such as VxWorks. Theoperating system 652 may include instructions for handling basic systemservices and for performing hardware dependent tasks.

The memory 650 may also store communication instructions 654 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. The memory 650 may includegraphical user interface instructions 656 to facilitate graphic userinterface processing; sensor processing instructions 658 to facilitatesensor-related processing and functions; phone instructions 660 tofacilitate phone-related processes and functions; electronic messaginginstructions 662 to facilitate electronic-messaging related processesand functions; web browsing instructions 664 to facilitate webbrowsing-related processes and functions; media processing instructions666 to facilitate media processing-related processes and functions;GPS/Navigation instructions 668 to facilitate GPS and navigation-relatedprocesses and instructions; camera instructions 670 to facilitatecamera-related processes and functions; and/or other messaginginstructions 672 to facilitate processes and functions, as described inreference to FIGS. 1-5.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures or modules. The memory 650 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the multi-touch capable device 400 may be implemented inhardware and/or in software, including in one or more signal processingand/or application specific integrated circuits.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The features can be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device or in a propagated signal, for executionby a programmable processor; and method steps can be performed by aprogrammable processor executing a program of instructions to performfunctions of the described implementations by operating on input dataand generating output.

The described features can be implemented advantageously in one or morecomputer programs that are executable on a programmable system includingat least one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that can be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program can be written in anyform of programming language (e.g., Objective-C, Java), includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors orcores, of any kind of computer. Generally, a processor will receiveinstructions and data from a read-only memory or a random access memoryor both. The essential elements of a computer are a processor forexecuting instructions and one or more memories for storing instructionsand data. Generally, a computer will also include, or be operativelycoupled to communicate with, one or more mass storage devices forstoring data files; such devices include magnetic disks, such asinternal hard disks and removable disks; magneto-optical disks; andoptical disks. Storage devices suitable for tangibly embodying computerprogram instructions and data include all forms of non-volatile memory,including by way of example semiconductor memory devices, such as EPROM,EEPROM, and flash memory devices; magnetic disks such as internal harddisks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a LAN, a WAN, and thecomputers and networks forming the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork. The relationship of client and server arises by virtue ofcomputer programs running on the respective computers and having aclient-server relationship to each other.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example,elements of one or more implementations may be combined, deleted,modified, or supplemented to form further implementations. As yetanother example, the logic flows depicted in the figures do not requirethe particular order shown, or sequential order, to achieve desirableresults. In addition, other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method performed at an electronic device with atouch-sensitive surface, the method comprising: receiving a documentfrom a server, the document including an embedded script; rendering anddisplaying the document at the electronic device and executing theembedded script, including: establishing a touchevent interface objectthat includes a plurality of touchlists; and upon detecting one or moretouches on the touch-sensitive surface: updating the toucheventinterface object with touch data, including values in two or more of thetouchlists; and further executing the embedded script in accordance withthe values in at least one of the two or more touchlists, wherein: thetouchevent interface object includes a plurality of distinct touchlists,a respective touchlist including two or more concurrent touches; andupdating the touchevent interface object with touch data includesupdating values in two or more of the touchlists.
 2. The method of claim1, wherein the touchevent interface object is for a first respectivedisplayed region of the document, the plurality of touchlists includes afirst touchlist listing all touches on the touch-sensitive surface and atarget touchlist listing only touches in the first respective displayedregion of the document.
 3. The method of claim 1, wherein the toucheventinterface object comprises a first touchevent interface object for afirst respective displayed region of the document; the method furtherincluding: establishing a second touchevent interface object for asecond respective displayed region of the document, the secondtouchevent interface object including a second plurality of touchlists;upon detecting the one or more touches, updating the second toucheventinterface object with touch data, including values in two or more of thesecond plurality of touchlists; and further executing the embeddedscript in accordance with the values in at least one of the secondplurality of touchlists; wherein the second plurality of touchlistsincludes a second touchlist listing all touches on the touch-sensitivesurface and a second target touchlist listing only touches in the secondrespective displayed region of the document.
 4. The method of claim 1,including: in accordance with an instruction in the embedded scriptcorresponding to a respective element of the document, establishing thetouchevent interface object as a touchevent interface objectcorresponding to the respective document element, the toucheventinterface object including the plurality of touchlists; and upondetecting one or more touches, updating the touchevent interface objectwith touch data, including values in two or more of the touchlists. 5.The method of claim 1, wherein the plurality of touchlists in thetouchevent interface object include a changetouch touchlist foridentifying one or more changed touches, the method further comprising:upon detecting a change in at least one of the one or more touches,updating the changetouch touchlist in the touchevent interface objectwith touch data identifying at least one changed touch; and furtherexecuting the embedded script in accordance with the updated changetouchtouchlist.
 6. The method of claim 1, wherein the plurality of touchlistsin the touchevent interface object include a targettouch touchlist foridentifying one or more touches on a target, the method furthercomprising: upon detecting a change in at least one of the one or moretouches on the target, updating the targettouch touchlist in thetouchevent interface object with touch data identifying at least onetouch on the target; and further executing the embedded script inaccordance with the updated targettouch touchlist.
 7. The method ofclaim 1, further comprising: upon detecting a change in at least one ofthe one or more touches, updating two or more of the plurality oftouchlists.
 8. An electronic device, comprising: a touch-sensitivesurface; one or more processors; and memory storing one or more programsfor execution by the one or more processors, the one or more programsincluding instructions for: receiving a document from a server, thedocument including an embedded script; rendering and displaying thedocument at the electronic device and executing the embedded script,including: establishing a touchevent interface object that includes aplurality of touchlists; and upon detecting one or more touches on thetouch-sensitive surface: updating the touchevent interface object withtouch data, including values in two or more of the touchlists; andfurther executing the embedded script in accordance with the values inat least one of the two or more touchlists, wherein: the toucheventinterface object includes a plurality of distinct touchlists, arespective touchlist including two or more concurrent touches; andupdating the touchevent interface object with touch data includesupdating values in two or more of the touchlists.
 9. The device of claim8, wherein the touchevent interface object is for a first respectivedisplayed region of the document, and the plurality of touchlistsincludes a first touchlist listing all touches on the touch-sensitivesurface and a target touchlist listing only touches in the firstrespective displayed region of the document.
 10. The device of claim 8,wherein the touchevent interface object comprises a first toucheventinterface object for a first respective displayed region of thedocument; and the one or more programs include instructions for:establishing a second touchevent interface object for a secondrespective displayed region of the document, the second toucheventinterface object including a second plurality of touchlists; upondetecting the one or more touches, updating the second toucheventinterface object with touch data, including values in two or more of thesecond plurality of touchlists; and further executing the embeddedscript in accordance with the values in at least one of the secondplurality of touchlists; wherein the second plurality of touchlistsincludes a second touchlist listing all touches on the touch-sensitivesurface and a second target touchlist listing only touches in the secondrespective displayed region of the document.
 11. The device of claim 8,wherein the one or more programs including instructions for: inaccordance with an instruction in the embedded script corresponding to arespective element of the document, establishing the toucheventinterface object as a touchevent interface object corresponding to therespective document element, the touchevent interface object includingthe plurality of touchlists; and upon detecting one or more touches,updating the touchevent interface object with touch data, includingvalues in two or more of the touchlists.
 12. The device of claim 8,wherein the plurality of touchlists in the touchevent interface objectinclude a changetouch touchlist for identifying one or more changedtouches; and the one or more programs include instructions for: upondetecting a change in at least one of the one or more touches, updatingthe changetouch touchlist in the touchevent interface object with touchdata identifying at least one changed touch; and further executing theembedded script in accordance with the updated changetouch touchlist.13. The device of claim 8, wherein the plurality of touchlists in thetouchevent interface object include a targettouch touchlist foridentifying one or more touches on a target; and the one or moreprograms include instructions for: upon detecting a change in at leastone of the one or more touches on the target, updating the targettouchtouchlist in the touchevent interface object with touch data identifyingat least one touch on the target; and further executing the embeddedscript in accordance with the updated targettouch touchlist.
 14. Thedevice of claim 8, wherein the one or more programs include instructionsfor: upon detecting a change in at least one of the one or more touches,updating two or more of the plurality of touchlists.
 15. Anon-transitory computer readable storage medium storing one or moreprograms for execution by one or more processors of an electronic devicewith a touch-sensitive surface, the one or more programs includinginstructions for: receiving a document from a server, the documentincluding an embedded script; rendering and displaying the document atthe electronic device and executing the embedded script, including:establishing a touchevent interface object that includes a plurality oftouchlists; and upon detecting one or more touches on thetouch-sensitive surface: updating the touchevent interface object withtouch data, including values in two or more of the touchlists; andfurther executing the embedded script in accordance with the values inat least one of the two or more touchlists, wherein: the toucheventinterface object includes a plurality of distinct touchlists, arespective touchlist including two or more concurrent touches; andupdating the touchevent interface object with touch data includesupdating values in two or more of the touchlists.
 16. The computerreadable storage medium of claim 15, wherein the touchevent interfaceobject is for a first respective displayed region of the document, andthe plurality of touchlists includes a first touchlist listing alltouches on the touch-sensitive surface and a target touchlist listingonly touches in the first respective displayed region of the document.17. The computer readable storage medium of claim 15, wherein thetouchevent interface object comprises a first touchevent interfaceobject for a first respective displayed region of the document; and theone or more programs include instructions for: establishing a secondtouchevent interface object for a second respective displayed region ofthe document, the second touchevent interface object including a secondplurality of touchlists; upon detecting the one or more touches,updating the second touchevent interface object with touch data,including values in two or more of the second plurality of touchlists;and further executing the embedded script in accordance with the valuesin at least one of the second plurality of touchlists; wherein thesecond plurality of touchlists includes a second touchlist listing alltouches on the touch-sensitive surface and a second target touchlistlisting only touches in the second respective displayed region of thedocument.
 18. The computer readable storage medium of claim 15, whereinthe one or more programs including instructions for: in accordance withan instruction in the embedded script corresponding to a respectiveelement of the document, establishing the touchevent interface object asa touchevent interface object corresponding to the respective documentelement, the touchevent interface object including the plurality oftouchlists; and upon detecting one or more touches, updating thetouchevent interface object with touch data, including values in two ormore of the touchlists.
 19. The computer readable storage medium ofclaim 15, wherein the plurality of touchlists in the toucheventinterface object include a changetouch touchlist for identifying one ormore changed touches; and the one or more programs include instructionsfor: upon detecting a change in at least one of the one or more touches,updating the changetouch touchlist in the touchevent interface objectwith touch data identifying at least one changed touch; and furtherexecuting the embedded script in accordance with the updated changetouchtouchlist.
 20. The computer readable storage medium of claim 15, whereinthe plurality of touchlists in the touchevent interface object include atargettouch touchlist for identifying one or more touches on a target;and the one or more programs include instructions for: upon detecting achange in at least one of the one or more touches on the target,updating the targettouch touchlist in the touchevent interface objectwith touch data identifying at least one touch on the target; andfurther executing the embedded script in accordance with the updatedtargettouch touchlist.
 21. The computer readable storage medium of claim15, wherein the one or more programs include instructions for: upondetecting a change in at least one of the one or more touches, updatingtwo or more of the plurality of touchlists.